Interdisciplinary Studies of Multisensory Integration: from Single Units to Perception

The integration of information from multiple sensory sources is a ubiquitous and highly adaptive process that can dramatically alter behavior and shape perception. Neurophysiological studies in animal models and in human subjects have begun to reveal the combinatorial rules by which neurons and networks combine this multisensory information to create a coherent perceptual unity. In addition, recent work has highlighted that the development of multisensory circuits is strongly shaped by early sensory experience, and has the capacity for significant plasticity. Most recently, studies have begun to focus on the plastic potential of adult multisensory processes. This work has found that significant plasticity can be engendered in adult multisensory systems, but only by coupling changes in the statistical relations of the combined stimuli with reinforcement. Such findings have significant implications for our understanding of basic integrative processes and have shed important light in bridging between neuronal and perceptual levels of analysis. An Introduction to Multisensory Processes We live in a world rich in sensory information. This information is conveyed via various forms of stimulus energy (e.g., chemical, mechanical, electromagnetic, thermal, etc.), and our sensory systems have evolved specialized peripheral organs to transduce these energies into a common neural code (i.e., action potentials). Heroic efforts in neuroscience research over the past 50 years have focused on understanding the nature of these transduction processes, and on elucidating the “neural code” for each of the sensory systems. Intriguingly, this work has largely focused on understanding these processes within the individual sensory systems, and as a result we have a detailed understanding of the mechanics of the transduction and encoding events within the visual, auditory, somatosensory, vestibular, gustatory and olfactory systems. However, intimate knowledge of these events in each of these sensory systems is ultimately inadequate for understanding the nature of our perceptual gestalt, since this unity is built also from the synthesis of information across the different senses. Numerous examples serve to highlight the powerful ability that the different sensory systems have to influence one another, and ultimately to shape our view of the world around us. One of the most entertaining of these examples is that of the ventriloquist, in which discordant visual cues (i.e., the movements of the dummy’s lips and head) can dramatically alter our judgments about the source of an auditory signal (i.e., the ventriloquist’s voice; see Thurlow and Jack 1973). In this particular example we refer to the ability of the visual cues to bias our localization of an auditory signal. In addition to these biasing effects under discordant conditions, visual cues can also dramatically improve the intelligibility of an auditory signal when presented in spatial concordance. For example, in a crowded and noisy room, the ability to see a speaker’s lip movements confers an effective 15 dB gain in our ability to comprehend their speech (Sumby and Pollack 1954). Finally, and as first described